The boosting: A new idea of building models

a r t i c l e i n f o The idea of boosting deeply roots in our daily life practice, which constructs the general aspects of how to think about chemical problems and how to build chemical models. In mathematics, boosting is an iterative reweighting procedure by sequentially applying a base learner to reweighted versions of the training data whose current weights are modified based on how accurately the previous learners predict these samples. By using different loss criteria, boosting copes with not only classification problems but also regression problems. In this paper, the basic idea and algorithms of commonly used boosting are discussed in detail. The applications to two datasets are conducted to illustrate the significant performance of boosting. Boosting, based on an ensemble of individual models, is one of the most powerful learning methods introduced in the last ten years. It stems from the PAC (probably approximately correct) learning framework [1] or the concept of ensemble learning [2,3] and was originally designed for classification problems. Based on the PAC theory, Kearns and Valiant [4] were the first to pose the question of whether a " weak " learning algorithm that performs just slightly better than random guessing can be " boosted " into an arbitrarily accurate " strong " learning algorithm. Such a question forms the foundation of boosting. The underlying idea of boosting is a procedure that combines the outputs of many " weak " learners to produce a powerful " committee'. A weak learner is an algorithm whose error rate is only slightly better than random guessing. Ensemble methods are learning algorithms that construct a set of classifiers and then classify new data points by taking a (weighted) vote of their predictions [5–7]. Nevertheless, it should be noted that the ensemble obtained through boosting can reduce the variance and bias of a model simultaneously. The first provable polynomial-time boosting algorithm which suffered from certain practical drawbacks was developed in 1990 by Schapire [8] in the PAC learning framework. The AdaBoost algorithm, introduced in 1995 by Freund and Schapire [9], solved many of the practical difficulties of the earlier boosting algorithms. AdaBoost, as the most popular boosting procedure, has since drawn much attention [10–16]. In order to find out why AdaBoost, in most cases, performs well, Friedman et al. [17,18] analyzed AdaBoost statistically, derived the exponential criterion, and showed that it estimated the log-odds of …